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Abstract. Ability to find and get services is a key requirement in the development of large-scale distributed sys- 
tems. We consider dynamic and unstable environments, namely Peer-to-Peer (P2P) systems. In previous work, we 
designed a service discovery solution called Distributed Lexicographic Placement Table (Dlpt), based on a hierar- 
chical overlay structure. A self-stabilizing version was given using the Propagation of Information with Feedback 
(PIF) paradigm. In this paper, we introduce the self-stabilizing CoPIF (for Collaborative PIF) scheme. An algo- 
rithm is provided with its correctness proof. We use this approach to improve a distributed P2P framework designed 
for the services discovery. Significantly efficient experimental results are presented. 

1 Introduction 

Computing abilities (or services) offered by large distributed systems are constantly increasing. Cloud envi- 
ronment grows in this way. Ability to find and get these services (without the need for a centralized server) is 
a key requirement in the development of such systems. Service discovery facilities in distributed systems led 
to the development of various overlay structures built over Peer-to-Peer (P2P) systems, e.g., II 121 181261271 . 
Some of them rely on spanning tree structures II 121271 . mainly to handle range queries, automatic completion 
of partial search strings, and to extend to multi-attribute queries. 

Although fault-tolerance is a mandatory feature of systems targeted for large scale platforms (to avoid 
data loss and to ensure proper routing), tree-based distributed structures, including tries, offer only a poor 
robustness in dynamic environment. The crash of one or more nodes may lead to the loss of stored objects, 
and may split the tree into several subtrees. 

The concept of self-stabilization [ 16] is a general technique to design distributed systems that can handle 
arbitrary transient faults. A self-stabilizing system, regardless of the initial state of the processes and the 
initial messages in the links, is guaranteed to converge to the intended behavior in finite time. 

In |[T0l . a self-stabilizing message passing protocol to maintain prefix trees over practical P2P networks 
is introduced. The protocol is based on self-stabilizing PIF (Propagation of Information with Feedback) 
waves that are used to evaluate the tree maintenance progression. The scheme of PIF can be informally 
described as follows: a node, called initiator, initiates a PIF wave by broadcasting a message m into the 
network. Each non-initiator node acknowledges to the initiator the receipt of m. The wave terminates when 
the root has received an acknowledgment from all other nodes. In arbitrary distributed systems, any node 
may need to initiate a PIF wave. Thus, any node can be the initiator of a PIF wave and several PIF protocols 
may run concurrently (in that case, every node maintains locally a data structure per initiator). 

Contribution. We first present the scheme of collaborative PIF (referred as CoPIF). The main thrust of this 
scheme is to ensure that different waves may collaborate to improve the overall parallelism of the mechanism 
of PIF waves. In other words, the waves merge together so that they do not have to visit parts of the network 
already visited by other waves. Of course, this scheme is interesting in environments were several PIF waves 
may run concurrently. Next, we provide a self- stabilizing CoPIF protocol with its correctness proof. To the 
best of our knowledge, it is the first self-stabilizing solution for this problem. Based on the snap-stabilizing 
PIF algorithm in (S), it merges waves initiated at different points in the network. In the worst case where only 



one PIF wave runs at a time, our scheme does not slow down the normal progression of the wave. Finally, 
we present experimental results showing the efficiency of our scheme use in a large scale P2P tree-based 
overlay designed for the services discovery. 

Roadmap. The related works are presented in Section [2] Section [3] provides the conceptual and computa- 
tional models of our framework. In Section [4j we present and prove the correctness of our self-stabilizing 
collaborative protocol. In Section[5J experiments show the benefit of the CoPIF approach. Finally, conclud- 
ing remarks are given in Section [6] 

2 Related Work 

2.1 Self-stabilizing Propagation of Information 

PIF wave algorithms have been extensively proposed in the area of self-stabilization, e.g., B2I5I8I14I30I 
to quote only a few. Except B5I14I30L all the above solutions assume an underlying self- stabilizing rooted 
spanning tree construction algorithm. The solutions in 1181 1411 have the extra desirable property of being snap- 
stabilizing. A snap-stabilizing protocol guarantees that the system always maintains the desirable behavior. 
This property is very useful for wave algorithms and other algorithms that use PIF waves as the underlying 
protocols. The basic idea is that, regardless of the initial configuration of the system, when an initiator starts a 
wave, the messages and the tasks associated with this wave will work as expected in a normal computation. 
A snap-stabilizing PIF is also used in [11] to propose a snap-stabilizing service discovery tool for P2P 
systems based on prefix tree. 

2.2 Resource Discovery 

The resource discovery in P2P environments has been intensively studied BUI . Although DHTs H24I25I281 
were designed for very large systems, they only provide rigid mechanisms of search. A great deal of research 
went into finding ways to improve the retrieval process over structured peer-to-peer networks. Peer-to-peer 
systems use different technologies to support multi-attribute range queries f6l 181261271 . Trie-structured ap- 
proaches outperform others in the sense that logarithmic (or constant if we assume an upper bound on the 
depth of the trie) latency is achieved by parallelizing the resolution of the query in several branches of the 
trie. 

2.3 Trie-based related work 

Among trie-based approaches, Prefix Hash Tree (PHT) Il22ll dynamically builds a trie of the given key-space 
(full set of possible identifiers of resources) as an upper layer mapped over any DHT-like network. Fault- 
tolerance within PHT is delegated to the DHT layer. Skip Graphs, introduced in 0, are similar to tries, 
and rely on skip lists, using their own probabilistic fault-tolerance guarantees. P-Grid is a similar binary 
trie whose nodes of different sub-parts of the trie are linked by shortcuts like in Kademlia ||T9l . The fault- 
tolerance approach used in P-Grid IPT51 is based on probabilistic replication. 

In our approach, the Dlpt was initially designed for the purpose of service discovery over dynamic 
computational grids and aimed at solving some drawbacks of similar previous approaches. An advantage 
of this technology is its ability to take into account the heterogeneity of the underlying physical network to 
build a more efficient tree overlay, as detailed in lfT3l . 

3 P2P Service Discovery Framework 

In this section we present the conceptual model of our P2P service discovery framework and the Dlpt data 
structure on which it is based. Next, we convert our framework into the computational model on which our 
proof is based. 



3.1 Conceptual Model 



3 



The two abstraction layers that compose our P2P service discovery framework are organized as follow: (i) a 
P2P network which consists of a set of asynchronous peer (physical machines) with distinct identifiers. The 
peer communicate by exchanging messages. Any peer PI is able to communicate with another peer P2 only 
if PI knows the identifier of P2. The system is seen as an undirected graph G = (V,E) where V is the set 
of peers and E is the set of bidirectional communication link; (ii) an overlay that is built on the P2P system, 
which is considered as an undirected connected labeled tree G' = (V',E') where V' is the set of nodes and 
E' is the set of links between nodes. Two nodes p and q are said to be neighbors if and only if there is a 
link (p,q) between the two nodes. To simplify the presentation we refer to the link (p,q) by the label q in 
the code of p. The overlay can be seen as an indexing system whose nodes are mapped onto the peers of 
the network. Henceforth, to avoid any confusion, the word node refers to a node of the tree overlay, i.e., a 
logical entity, whereas the word peer refers to a physical node part of the PIP system. 

Reading and writing features of our service discovery framework are ensured as follow. Nodes are 
indexed with service name and resource locations are stored on nodes. So, client requests are treated by 
any node, rooted to the targeted service labeled node along the overlay abstraction layer, indexed resource 
locations are returned to the clients or updated . A more detailed description of the implementation of our 



framework is given in (9J and briefly reminded in Section 5. 1 



The Distributed Lexicographic Placement Table (Dlpt H12I13I0 is the hierarchical data structure that 
ensures request routing across overlay layer. Dlpt belongs to the category of overlays that are distributed 
prefix trees, e.g., H4I23I11 Such overlays have the desirable property of efficiently supporting range queries 
by parallelizing the searches in branches of the tree and exhibit good complexity properties due to the limited 
depth of the tree. More particularly, Dlpt is based on the particular Proper Greatest Common Prefix Tree 
(PGCP tree) overlay structure. A Proper Greatest Common Prefix Tree (a.La radix tree in ETI ) is a labeled 
rooted tree such that the following properties are true for every node of the tree: (i) the node label is a proper 
prefix of any label in its subtree; (ii) the greatest common prefix of any pair of labels of children of a given 
node are the same and equal to the node label. 

Designed to evolve in very dynamic systems, the Dlpt integrates a self-stabilization mechanisms iPTOl . 
providing the ability to recover a functioning state after arbitrary transient failures. As such, the truthfulness 
of information returned to the client needs to be guaranteed. We use the PIF mechanism to check whether 
Dlpt is currently in a recovering phase or not. 



3.2 Computational Model 

In a first step, we abstract the communication model to ease the reading and the explanation of our solution. 
We assume that every pair of neighboring nodes communicate in the overlay by direct reading of variables. 
So, the program of every node consists in a set of shared variables (henceforth referred to as variables) and 
a finite number of actions. Each node can write in its own variables and read its own variables and those of 
its neighbors. Each action is constituted as follow: < Label >::< Guard >— X Statement >. The guard of 
an action is a Boolean expression involving the variables of p and its neighbors. The statement is an action 
which updates one or more variables of the node p. Note that an action can be executed only if its guard 
is true. Each execution is decomposed into steps. Let y be an execution and A an action of p (p G V). A is 
enabled for p in y if and only if the guard of A is satisfied by p in y. Node p is enabled in y if and only if at 
least one action is enabled at p in y. 

The state of a node is defined by the value of its variables. The state of a system is the product of the 
states of all nodes. The local state refers to the state of a node and the global state to the state of the system. 
Each step of the execution consists of two sequential phases atomically executed: (i) Every node evaluates 
its guard; (ii) One or more enabled nodes execute their enabled actions. When the two phases are done, the 
next step begins. 
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Fig. 1. The PIF Wave. 



Formal description (Section 4.2 1 and proof of correctness of the proposed collaborative propagate in- 
formation feedback algorithm will be done using this computational model. Nevertheless, experiments are 
implemented using the classical message-passing model over an actual peer-to-peer system H7I17II . 



4 Collaborative Propagation of Information with Feedback Algorithm 

In this section, we first present an overview of the proposed Collaborative Propagation of Information with 
Feedback Algorithm (CoPIF). Next, we provide its formal description. 



4.1 Overview of the CoPIF 

Before explaining the idea behind CoPIF, let us first recall the well-known PIF wave execution. Starting 
from a configuration where no message has been broadcast yet, a node, also called initiator, initiates the 
broadcast phase and all its descendant except the leaf participate in this task by sending also the broadcast 
message to their descendants. Once the broadcast message reaches a leaf node of the network, they notify 
their ancestors of the end of the broadcast phase by initiating the feedback phase. During both broadcast 
and feedback steps, it is possible to collect information or perform actions on the entire data structure. Once 
all the nodes of the structure have been reached and returned the feedback message, the initiator retrieve 
collected information and executes a special action related to the termination of the PIF-wave. In the sequel, 
we will refer to this mechanism as classic-PIF. 

However, the PIF mechanism is a costly broadcast mechanism that involves the whole platform. In this 
paper, we aim to increase the parallelism of the PIF by making several PIF waves collaborating together. Let 
us now define some notions that will be used in the description of our solution: 

Let an ordered alphabet A be a finite set of letters. Lets define -< an order on A. A non empty word w 
over A is a finite sequence of letters a\, ... , a;, a/ such as / > 0. The concatenation of two words u and v, 
denoted as uv, is equal to the word a\, a,-, a^, by, bj, b t such that u = a\, at, a^ and v = b\, 

bj, b;. A word u is a prefix (respectively, proper prefix) of a word v if there exists a word w such 
that v = uw (respectively, v = uw and u ^ v). The Greatest Common Prefix (respectively, Proper Greatest 
Common Prefix) of wy and W2, denoted GCP{wy,W2) (respectively PGCP(w i,W2), is the longest prefix u 
shared by wy and W2 (respectively, such that V/ > 1, u ^ w{). 

Let us now describe the outline of the proposed solution through the P2P framework use case. 



Use Case. The idea of the algorithm is the following: When a user is looking for a service, it sends a request 
to the Dlpt to check whether the service exists or not. Once the request is on one node of the Dlpt, it is 



routed according to the labelled tree in the following manner: let l request be the label of the service requested 
by the user and let l p be the label of the current node u p . In the case of PGCP{l request ,lp) is true, u p checks 
whether there exists a child u q in the Dlpt having a label l q such that PGCP(l request ,l q ) is satisfied. If such a 
node exists, then u p forwards the request to its child u q . Otherwise (PGCP(l request ,l p ) is not satisfied), if we 
keep exploring the sub-tree routed in u p , the service will not be found. u p sends in this case the request to 
its father node in the Dlpt. By doing so, either (?) the request is sent to one node u p such that l p = l re questi 
or (if) the request reaches a node u p such that it cannot be routed anymore. In the former case, the service 
being found, a message containing the information about the service is sent to the user. In the latter case, the 
service has not been found and the message "no information about the service" is sent to the user. However, 
the node has no clue to trust the received information or not. In other words, in the former case, u p does not 
know whether it contains the entire service information or if a part of the information is on a node being at 
a wrong position in the tree due to transient faults. In the latter case, u p does not know whether the service 
is really not supported by the system or if the service is missed because it is at a wrong position. 

In order to solve this problem, u p initiates a PIF wave to check the state of all the nodes part of the Dlpt. 
Note that several PIF waves can be initiated concurrently since many requests can be made in different parts 
of the system. The idea of the solution is to make the different PIF waves collaborating in order to check 
whether the tree is under construction or not. For instance, assume that two PIF waves, PIF I and PIF2, are 
running concurrently on two different parts of the tree, namely on the subtrees Tl and T2, respectively. Our 
idea is to merge PIF\ and PIF2 so that PIF I (respectively, PIF2) do not traverse T2 (resp., Tl) by using 
data collected by PIF 2 (resp.WFl). Furthermore, our solution is required to be self-stabilizing. 

CoPIF. Basically, the CoPIF scheme is a mechanism enabling the collaboration between different PIF 
waves. Each node u p of the Dlpt has a state variable S p that includes three parameters S p = (Phase, id/, idpjp) 
Parameter idpiF refers to the identifier of the PIF wave which consists of the couple (id peer ,l Ui ), where id peer 
is the identifier of the peer hosting the node w, that initiated the PIF wave and l Uj is the label of the node w ; . 
The value id/ refers to the identifier of the neighbor from which u p received the broadcast. It is set at NULL 
in the case u p is the initiator. Phase can have four values: C, B, FC and FI. The value C (Clean) denotes 
the initial state of any node before it participates in a PIF wave. The value B (Broadcast) or FC (Feedback 
correct) or FI (Feedback incorrect) means that the node is part of a PIF wave. Observe that in the case 
there is just a single PIF wave that is executed on the Dlpt, then its execution is similar to the previously 
introduce classic-PTF. 

When more than one PIF wave are executed, four cases are possible while the progression of the CoPIF 
wave. First (i), if there is a node u p in the C state having only one neighboring node q in the B state and 
no other neighboring node in the FI or FC state, then p changes its state to B. Second (if), if there exists a 
leaf node u p in the C state having a neighbor u q in the B state, then u p changes its state to FC (resp. FI) if 
its position in the Dlpt is correct (resp. incorrect). Next (Hi), if there is a node u p in C-phase having two 
neighboring nodes u q and u q i in the B state with different idpiF then, u p changes its state to B and sets its 
id/ to u q such that the idpip of u q is smaller than idpip of u q >. Finally (iv), if there exists a node u p that is 
already in the B state such that its idf is u q and there exists another neighboring node u q i which is in the B 
state with a smaller idpip and a different idf, then u p changes its father by setting idf at u q >. 

Notice that in the fourth cases, u q (previously, the idf of u p ) will have to change its idf as well since it 
has now a neighbor u p in the B state with a smaller id peer . By doing so, the node w, that initiated a PIF wave 
with a smaller id will change its id peer . Similarly, notice that m; is not an initiator anymore. Hence it changes 
its idf from NULL to the id of its neighbor with a smallest idpiF- So, only one node will get the answer (the 
feedback of the CoPIF), this node being the one with the smallest idpip. Therefore, when an initiator sets 
its idf to a value different from NULL (as w; previously), it sends a message to the new considered initiator 
(can be deduced from idpip) to subscribe to the answer. So, when an initiator node receives the feedback 
that indicates the state of the tree, it notifies all its subscribers of the answer. 
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Fig. 2. 4 CoPIF wave transition. 



4.2 Formal Description 

In the following we first define the data and variables that are used for the description of our algorithm. We 
then present the formal description in Algorithm [T] 

- Predicates 

• Requestpip : Set at true when the peer wants to initiate a PIF wave (There is a Service — Request 
which could not find the desired service). 

- Variables 

• S p = (A,q, q') : refers to the state of the node p such that: A corresponds to the phase of the PIF wave 
p is in. A G {B,FI,FC,C} for respectively Broadcast, Feedback State-Incorrect, Feedback State- 
Correct, Clean, q refers to the identity of the peer that initiates the PIF wave, q' refers to the identity 
of the neighboring node of p in the Dlpt from which p got the Broadcast. 

• N„: refers to the set of the identities of the nodes that are neighbor to p 

• StateoLPT- refers to the state of the Dlpt 

• min p : q G N p , S q = (B,id q ,z) A Z^p A id p = min{id q i, q' G N p , Sq> = (B,id q >,z') A z! ^ p}. 

- Functions 

• Send(@dest,@source, Msg): ©source sends the message Msg to @dest. 

• Add(Mylist, item): Add to my list the subject item. 

Character ' - ' in the algorithm means any value. 

4.3 Correctness Proof. 

In the following, we prove the correctness of our algorithm. 
Let us first define our self-stabilizing CoPIF wave: 

Definition 1. (CoPIF wave) 

A finite computation e is called CoPIF wave if and only if the following conditions hold: 

— Each node in the system is able to initiate a PIF wave in a finite time. 

— All the nodes of the system are visited by CoPIF wave. 

— Exactly one node of the system (that initiated a PIF wave) receives the acknowledgement from all the 
other nodes. 

Let us now define some notions that will be used later. 
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Algorithm 1 CoPIF 

- PIF Initiation 

• Rl: RequestpiF A S p = (C, — . — ) A V q G N p , S q =fc (— , — ,p) — > S p = (B, (id pe „,l p ),NULL), StateoLPT = Unknown 

• R2: Requestpif A S p = (B,id,q) A q^L NULL — > Send(@id,idpeer,lnterested), StateotPT = Unknown 

- Broadcast propagation 

• R3: = (C,-,-) A -^Requestpw A 3 ? G jV p , (S, = (B,*, -) A ? = («i» f A ^3 9' G AT p , ? ^ ^, S,< = (B,k, -)) A V 9" G iVp, S^/ ^ 
(FIVFC,-,p)^S p = (B,k,q) 

- Father-Switch 

• R4: 3 q G /V p , S p = (B,id,q) A3q' e N p , (q + q' A = (BJd',7) A id' < id A q = min p )^S P = (B,id',q') 

- initiator resignation 

• R5: S p = (B, id, NULL) A 3 q G N p , S q = (B,id',l) A id' < id A q = min p ) -> S p = (B,id',q), Send{@id,id' peer, Interested) 

- Feedback initiation 

• R6: |JVp| = 1 A State = Correct A 3 q G N p , S q = (B, id, ?) -> S p = (FC, id, q) 

• R7: \N P \ = 1 A State = Incorrect A3qeN p ,S q = (B,id,l) -> Sp = (FI,id,q) 

- Feedback propagation 

. R8: 3 <? G A?p, S„ = (B,id,—) A S p = {B,-,q) A V </ G VM- S « = -,p) -^S„ = (FC,id,q) 

. R9: 3 q G AT,, S, = (B, id, -) A S p = (B,-,q) A V <?' € ^ = {FIVFC,-,p) A3q"e N„/{q} S' q = (FI, 1,p) -> S p = (F/, 

- Cleaning phase initiation 

• RIO: ¥ j E iV p , S 9 = (FC,id',p) A S p = (B, id, NULL) A id = (id peer ,l p ) — > StateotPT = Correct, Requestpip = false, S p = 
(C, NU LL, NULL) , Send(@ListToContact, 'DLPT Correct'), State DLPT = Unknown 

• Rll: V qeN p ,S q = (FIVFC, id! ,p) A S p = (B, id, NULL) A id = (id peer , l p ) A 3 q" G N p /{q] S' q = (FI,-,p) -> State DLPT ^Incorrect, 
Requestpip = false, S p = (C, NULL, NULL), Send(@ListToContact, 'DLPT Incorrect'), State DLPT = Unknown 

- Cleaning phase propagation 

• R12: 3qeN p , S p = (FIVFC, id,q) A (S q = (C,-, -) V q = NULL) -> S p = (C, NU LL, NULL) 

- Correction Rules 

• R13: Sp = (B, id, NULL) A id f (idpeer, l p ) -> S p = (C, NU LL.NULL) 

• R14: 3qeN p S p = (FIVFC,id,q) A 3 q' G N p , q f q' A + (FIVFC,-,-) -^S p = (C, NULL, NULL) 

• R15: Sp = (B,id,q) A (S q + (B, — ,—) V [S q + (B.id 1 ,-) A id 1 > id] -^S p = (C, NULL, NU LL) 

• R16: Sp = (B,—,q) A S q = (FIV FC,—,—) — > S p = (C, NU LL.NULL) 

• R17: 3q G N p , S p = (B,id,q) A S q = (B,id,p) -> S P (C,NULL,NULL) 

• R18: 3q,q' eN p ,S p = (B, id, q) A q + q 1 A S ? / = (B, id, z) Az + p^t S p = (F, id, q) 

• R19: 3q,q' e N p , S p = (C, NU LL, NULL) A S q = (B,id,z) Az^pAS q ,= (B,id,zf) A z' + p -> S p = (F,id,q) 

• R20: 3 q,q' G N p , S p = (B,id,q) A S q = (B,id',z) Az + p Aid' < id A S q , = (B,id",z r ) A z' + p A id' < id" -> S p = (B,id',q) 

- Event: Message reception 

• Message 'idpeer,Interested': Add(ListToContact, idpeer) 

• Message 'Contact id for an answer': Send(@id,idpeer,Interested) 

• Message 'DLPT Correct': StateotPT = Correct 

• Message 'DLPT Incorrect': StateotPT = Incorrect 



definition 2. ( abnormal sequences of type A ) 

We say that a configuration contains an abnormal sequence of type A if there exists a node p of state 
S p = (B,id,q) such as one of these conditions holds: 

1. q = NULL A id / (id peer ,lp). 

2. q^ NULL A S q = (B, id',z) A id' > id Az^p. 

3. q^ NULL A S q = (B, id, p). 

4. q ^ NULL A S q ^ (B, — , — ). 

In the following we refer to each case by type Ai where 1 < i < 4. 
Definition 3. (abnormal sequences of type B) 

We say that a configuration contains an abnormal sequence of type B if there exists a node p of state 
S p = (C, NULL, NULL) such as 3q,q' G N p , S q = (B,id,z) A S q > = (B,id,z') A z / p A z' / p. 

Definition 4. (Dynamic abnormal sequence) 

We say that a configuration contains a dynamic abnormal sequence if there exists two nodes p and q such 
as S p = (B, id,z) with z / q and S p = (B, id,z') with z' ^ p. 

Definition5. (Trap sequence) 

We say that a configuration contains a trap sequence if there exists a sequence of nodes po, p\, ■ Pk such 
that the following three conditions hold: (?) S Po = (B,id,z) A z / p\. (ii) S Pk = (B,id,z') A z / Pk-i- ("0 V 
1 < i < k- 1, S Pi = (C, NULL, NULL). 

Definition 6. (Path) 

The sequence of nodes = po, p\, p^ is called a path ifVl<i<k, S Pi = (B,id,pi-\) A S Po = 
(B, id, NULL), po is said to be the extremity of the path. 

Definition?. (FullPath) 

For any node p such as S p ^ (C, NULL, NULL), a unique path PN = po, p\, pk is called FullPath, if and 
only if V 1 <i<k, S Pi = (B V F, — , i ) A S Po = (fi V F,—,p). p is said to be the extremity of the path. 

Definition8. (SubTree) 

For any node p, we define a set of SubTree(p) of nodes as follow: for any node q, q £ SubTree(p) if and 
only if p and q are part of the same FullPath such as p is the extremity of the path. 

In the following, we say that a node p clean its state if it updates its state to S p = (C, NULL, NULL). 
Let us first show that a configuration without abnormal sequences of type A and B is reached in a finite 
time. 

Lemma 1 No abnormal sequence of type A can be created dynamically. 
Proof. Let consider each abnormal sequence of type A separately: 

1. Type Al. Note that the only rule in the algorithm that allows p to set its state to S p = (B, id, NULL) is/?l. 
Note also that when R\ is executed on p, id = (id peer ,lp). Thus we are sure that the abnormal sequence 
Al is never created dynamically. 

2. Type A2. Note that the only case where p sets its state to S p = (B,id,q) such as q / NULL is when 
S q = (B, id',z) such as id' < id and z^p (see Rules R3, R4 and R5). Thus we are sure that no abnormal 
sequence of type A2 is created dynamically. 



3. Type A3. To create this abnormal sequence dynamically either p or q or both are in the clean state* 
(C, NULL, NULL). In the three cases the node that has a clean state (C, NULL, NULL) (let this node 
be p) never change its state to set it to the broadcast phase (B,id,q) when S q = (C, NULL, NULL) V 
S q = (B, id,p). Thus we are sure that no abnormal sequence of type A3 is created dynamically. 

4. Type A4. The properties of the PIF algorithm ensure that when a node p is in a broadcast phase (B, id,q) 
(refer to (U), then S q = (B, — ,—). Thus we are sure that no abnormal sequence of type A4 is created 
dynamically. 

□ 

From Lemmas [TJ we can deduce that the number of abnormal sequences does not increase. 

Lemma 2 Every execution of Algorithm^contains a suffix of configurations containing no abnormal se- 
quence of type Al. 

Proof. Note that in a case of an abnormal sequence of type Al, there exists a node p whom state 
Sp = (B,id,q) such as q = NULL A id ^ (id peer ,lp). Note that R\3 is enabled on p. When the rule is 
executed on p, S p = (C, NULL, NULL) and the Lemma holds. □ 

Lemma 3 Every execution of Algorithm^contains a suffix of configurations containing no abnormal se- 
quence of type A2. 

Proof. Note that in a case of an abnormal sequence of type A2, there exists a node p whom state 
Sp = (B,id,q) such as q^NULL AS q = (B,id',z) A id' > id A z / p. Note that/?15 is enabled on p. When 
the rule is executed S p = (C, NULL, NULL) and the Lemma holds. □ 

Lemma 4 Every execution of Algorithm^contains a suffix of configurations containing no abnormal se- 
quence of type A3. 

Proof. Note that in a case of an abnormal sequence of type A3, there exists a node p whom state 
Sp = (B,id,q) such as q^NULL AS q = (B,id,p). Note that Rll is enabled on p. When the rule is executed 
S p = (C, NULL, NULL) and the Lemma holds. □ 

Lemma 5 Every execution of Algorithm^contains a suffix of configurations containing no abnormal se- 
quence of type A4. 

Proof. Let S = n\,n2, ...,«yt be the sequence of node on the tree overlay such as S n{ = (B,—,p) and 
S nj = (B, ). Note that in a case of an abnormal sequence of type A3, there exists a node p whom state 

Sp = (B,id,q) such as q / NULL A S q ^ (B, — ,— ). In this case R15 is enabled on p. When the rule is 
executed S p = (C, NULL, NULL). Note that the state of n\ is the same as p is the previous round, thus when 
n\ executes R\5, S, h = (C, NULL, NULL). R15 becomes then enabled on «2 and so on. Thus we are sure to 
reach in a finite time a configuration without any abnormal sequence of type A4. □ 

Lemma 6 Every execution of Algorithm^contains a suffix of configurations containing no abnormal se- 
quence of type B. 

Proof. Note that in the case of an abnormal sequence of type B there is in the configuration at least 
one node p such that the following properties hold: S p = (C,NULL,NULL) A 3 q,q' £ N p , S q = (B,id,z) 
A S q ' = (B,id,z r ) A z ^ p A z' ^ p. Note that in this case R19 is enabled on p. When the rule is executed, 



i^, = (F,id,q) (the scheduler will make the choice between q and q'). When all the nodes on which R\9 is 
enabled execute R\9. A configuration without any abnormal sequence of type B is reached and the lemma 
holds. □ 



Let us show now that a configuration without dynamic abnormal sequences is reached in a finite time. 

Lemma 7 If the configuration contains a trap sequence, then this sequence was already in the system in the 
starting configuration. 

Proof. The proof is by contradiction. We suppose that the trap sequence can be created during the 
execution using the PIF waves that were initiated after the starting configuration. 

First of all note that since the identifier of each PIF wave is unique, there is only one initiator for each PIF 
wave. Thus, there is at most one PIF wave that can be executed for each id (considering only the PIF waves 
that were executed after the initial configuration). On the other hand, when a PIF wave is executed, the only 
rules that makes a node change its father id are RA and R5. Note that when one of these rules is enabled 
on p, p has a neighboring node q in the broadcast phase with a smaller PIF id. When one of these rules is 
executed, p changes both the identifier of its father and the identifier of the PIF wave. Thus we are sure that 
if the configuration contains a trap sequence then, this sequence was already in the system in the starting 
configuration. □ 



Lemma 8 No dynamic abnormal sequence is created dynamically eventually. 
Proof. The two cases bellow are possible: 

1. There is a node p that has a clean state S p = (C, NULL, NULL) such as it has two neighboring nodes 
q and q' with S q = (B,id,z), S q i = (B,id,z'), p and z' / p- In this case R19 is enabled on p. When 
the rule is executed, p sets its state directly to the feedback phase S p = (F, id, q V q') (the adversary will 
choose between q and q'). Thus we are sure that no dynamic abnormal sequence is reached in this case. 

2. There are two neighboring nodes p and p' such as the two following condition hold: (/) 3q £ N p , 
S q = (B,id,z) A z i= p and (if) 3q' G N p >, S CJ > = {B,id,z') A z' i=- p'. Note that on both p and p', R3 
is enabled. When the rule is executed only on one node, we retrieve Case[T] Thus, no dynamic abnormal 
configuration is reached. In the case R3 is executed on both p and p' then a dynamic abnormal sequence 
is created. However, observe that the latter case cannot happen infinitely often since in a correct execu- 
tion, it is impossible to reach a configuration where two PIF waves with the same id are executed on 
two disjoint sub-trees (refer to Lemma [7]). So we are sure that a limited number of dynamic abnormal 
sequences can be created (due to the arbitrary starting configuration). Thus we are sure that, after a finite 
time, no dynamic abnormal configuration is reached. 

From the cases above, we can deduce that no dynamic abnormal sequence is created dynamically eventually 
and the lemma holds. □ 



Lemma 9 Every execution of Algorithmic ontains a suffix of configurations containing no dynamic abnor- 
mal sequences. 

Proof. From Lemma [8] a limited number of dynamic abnormal sequences can be created. Let's con- 
sider the system when all the dynamic abnormal sequences have been created (no other Dynamic abnormal 
sequence can be created). Note that when the configuration contains a dynamic abnormal sequence, there 
are at least two nodes p and q that are neighbors such as S p = (B,id,z), S q = (B,id,z'), z ^ q and z' 7^ p. 
Note that R 18 is enabled on both p and q. When the rule is executed on at least one of the two nodes (let this 
node be the node p),S p = (F, id,z) and the lemma holds. □ 



From the lemmas above we can deduce that a configuration without any abnormal sequences of (type A 
and B), any dynamic abnormal sequences is reached in a finite time. 

In the following we consider the system when there are no abnormal sequences (Dynamic, Type A and 
B). We have the following lemma: 

Lemma 10 Let P^ = po, pi, p^ be a path. If there exist 1 < i < k such as pi changes its state to S Pi = 
(B,id',q) with q / and id' < id then, V < j < i, pj will also updates their state to S p . = {B,id' ,pj + \) 
in a finite time. 

Proof. Note that when pi changes its state to S Pi = (B,id' ,q) with q ^ pi-\ and id' < id, becomes 
neighbor of a node (pi) that is in the broadcast phase with a smallest id, pi-\ changes its state to 5p._, = 
(B,id' ,pt) by executing R4. Note that now R15 now is enabled on and so on, thus we are sure that V 
1 < 7 < i, Pj will updates their state to S Pj = (B,id' ,pj + i). Note that when p\ updates its state, R5 becomes 
enabled on po, when R5 is executed, po = (B, id,pi). Thus the lemma holds. □ 

Let us refer by Initial — PIF waves, the set of PIF waves that were already in execution in the initial 
configuration (they were not initiated after the faults). Recall that the starting configuration can be any 
arbitrary configuration. Thus, some PIF waves can be separated by nodes that are in the feedback phase: 
PIFtf and PIF^i are said to be separated by nodes in the feedback phase if there exists a sequence of nodes 
po,pi,...,Pk such that S Po = (B,id,z) with z^p\ and S Pk = (B,id',z') with z' / Pk-i and (S Pl = (F,id,po) 
V S p _ l = (F,id' ,Pk))- Let refer by PIF friendly set 5,-, the set of nodes that are part of PIF waves that are 
not separated by nodes in the feedback phase. Note that 1 < i < k. 

In the following, we say that there is a Partial-Final sequence in a PIF friendly set Sj, if there exists a 
node p such that S p = (B, id, NULL) and V q G Sj/{p}, q G SubTree(p) and S q = (F,—, — ) (refer to Figure 

0- 




We can state the follow lemma: 
Lemma 11 In every PIF friendly set, Partial-Final sequence is reached infinite 

Proof. Let consider a single PIF friendly set 5,-. It is clear that if in Si there is only one PIF wave that 
is executed, then the lemma holds since the PIF wave in Sj behaves as the PIF wave described in [8]. In the 
following we suppose that there are at least two PIF waves that are executed in each PIF friendly set. 

Observe that the behavior of PIF waves when they are alone (when they do not meet any other PIF 
wave) is similar to the well now PIF schema in [8], thus, we will not discuss their progression in this case. 
Let us consider the PIF wave that has the smallest identifier id in the set S, (let refer to its initiator by 
InitTarget). The cases bellow are then possible: 



1 f. There exists a node p such as S p = (C, NULL, NULL) A 3 q,q' G N p , S q = (B,id,z) A = (B,id',z r ). 
Note that id < id'. In this case p sets its state to the broadcast phase and chooses q as its father S p = 
(B, id,q). Note that for the node q' we retrieve case [2] 

2. There exists a node p such as S p = (B, id' ,q) A 3 q' £ _/V p , = (B,id,z) Az / p. Note that id < id'. In 
this case we are sure that S 9 = (B, id" ,z) with id" < id' and z 7^ p (recall that there in no more abnormal 
sequences). Thus, p changes its state to S p = (B,id,q'). Note that p was part of path Py. From Lemma 
[TO} all the nodes on the path will updates their state as well. Thus the initiator of the PIF wave of 
identifier id' will be able to know that there is another PIF wave with a smaller id that is being executed 
(let refer to such a node by Init^i). Init it j> updates it state to be part of the PIF wave of identifier id (refer 
to Rule R5). Note that only the node that are on the path of both p and Initio updates their state. The 
other nodes (that were part of PIF^i) don't have to update their state since when the nodes of the path 
update their state, they are already part of SubTree(p). 

Observe that since some of the nodes that are part of SubTree(p) do not change their state, Some PIF 
waves can be seen as PIF waves with smaller id. Refer to Figure [4j (From the figure (case (c)) we can 
observe that p' is not aware of the presence of the PIF wave of id = since it did not change its state). 
Let po be the node that is neighbor to the initiator of the PIF wave PIF 1 that has the smallest id. Note 
that S Po = (B,id,z) (p in Figure |4| and let pk be the node that is neighbor to the initiator of the other 
PIF (the one that can be considered as the PIF wave with the smallest id, the PIF wave with id = 1 in 
Figure[4]). Observe that S Pk = (B,id',z'). Let P = Pi,P2,P3,---Pk-i be the sequence of nodes between 
po and pk such as S Pi = (B,id" ,p/-i) and id" > id. Note that on Pk-i, R4 is enabled. When the rule 
is executed, pk-\ updates its state to S Pk _ x = (B,id',pk). Note that on pk-i, R4 becomes enabled thus, 
Pk-2 will have the same behavior, it updates its state to S Pk 2 = (B,id' ,Pk-i) and so on. Hence, all the 
nodes on P except p\ will updates their state and set it at (B,id' ,p ;+ i). Note that p\ is able to detect the 
presence of the two PIF waves since it has two neighboring nodes that are part of different PIF waves, 
it is able to detect that P2 think that the PIF with the identifier id' is the smallest one. Thus, p\ updates 
its state by changing the identifier of the PIF waves to set it at the smallest one. (S Pl = (B,id,z)). By 
doing so, P4 becomes enabled on pi. When the rule is executed, P2 updates its state and so on. 

Let pMt be the node that initiates a PIF wave that has the smallest id within the PIF friendly 5,-. From the 
two cases above, we can deduce that all the nodes part of the same 5, will be part of the SubTree(pi n j t ). Note 
that when the feedback phase finished its execution, all the nodes in Sj except p !m y will be in the feedback 
phase. Thus a Partial-Final configuration is reached and the lemma holds. □ 

From the two cases above, we can deduce that after a finite time, a final-partial configuration is reached 
in a finite time and the lemma holds. 

Lemma 12 Every node belonging to a PIF friendly set eventually clean its state. 

Proof. Note that when the configuration of type partial-Final is reached in each set, there is only 
one node p that is enabled. Observe that S p = (B, id, NULL) (refer to the Rules P10 and Pll). When p 
executes P10, it updates its state to S p = (C, NULL, NULL). Let Ni be the set of nodes that are neighbor of 
p. Recursively, let Ni be the set of nodes that are neighbor of one node that is part of iV ; -_i. Note that P12 
becomes enabled on all the nodes part of SI, when the rule is executed, each node clean its state. In the same 
manner, PI 2 becomes enabled on the nodes part of 52, and so on. Thus we are sure all the nodes part of each 
PIF friendly set will eventually clean their state and the lemma holds. 

□ 



From Lemma 12 all the nodes part of Initial — PIF waves clean their state in a finite time. Thus we are 
sure that a configuration without any Initial — PIF wave is reached in a finite time. 



Initiator 




(a) A PIF wave with id=3 is being executed. (b) a PIF wave with id = is initiated, p updates its state. 




the smallest one and updates its state. (d) q' notices the situation and updates the idpjf. 

Fig. 4. Special Case. 



Let us now consider the system at that time (without Initial — PIF waves). In the following we extend 
the notion of Partial-Final configuration as follow: we say that a configuration is of type Final-Configuration 
if there exists in the system a single PIF friendly set such that there exists a node p that verifies the following 
condition: S p = (B, id, NULL) and V q G Si/{p}, q G SubTree(p) and S q = (F,—,—) 

The following lemma follows: 

Lemma 13 If there are many PIF waves that are executed then after a finite time, Final Configuration is 
reached. 

Proof. Note that since all the PIF waves that are on the system were initiated by nodes, there is exactly 
one PIF friendly set i.e., there are no PIF waves that are separated by nodes in the feedback phase (since 
the Feedback phase is initiated from the leaves of the tree overlay and a node p is allowed to change its to 
the feedback phase only if all its neighboring nodes except his father are already in the feedback phase). 
Observe that all the nodes of the system are part of the PIF friendly set. Observe also that the partial-Final 
configuration in this case is exactly the same as the Final configuration. Thus, we can deduce from Lemma 
1 1 that Final configuration is reached in a finite time, and the lemma holds. □ 



Lemma 14 Starting from a final configuration, all the nodes of the system eventually clean their state. 

Proof. Can be deduce directly from Lemma [T2| (since Partial-Final configuration is the same as Final 
configuration when there is a single PIF friendly set in the system). Thus, the lemma holds. □ 

Theorem 1. Every node is infinitely often able to initiate a PIF wave 



Proof. Directly follows from Lemma 14 and the fact that if Rule Rl becomes enabled on p, then it 
remains enabled until R\ is executed — R3 cannot be enabled while RequestpiF = true. 
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□ 



We can now state the following result: 
Theorem 2. Algorithm 1 is a self-Stabilizing CoPIF algorithm. 



Proof. From Theorem [T] each node is able to generate a PIF waves in a finite time. From Lemma 13 



all the nodes of the system were visited by the CoPIF wave. Thus, all of them acknowledge the receipt of 
the question (whether the tree overlay is in a correct state or not) and give an answer to the latter. Finally, 
from Lemma 13 one node p of the system receives the answer (S p = (B, id, NULL)). Hence we can deduce 



that Algorithm 1 is a self-Stabilizing CoPIF algorithm and the theorem holds. □ 



5 Evaluation 



In order to evaluate qualitatively and quantitatively the efficiency of CoPIF, we drive a set of experiments. 
As mentioned before, the Dlpt approach and its different features have been validated through analysis and 
simulation |[29l . The scalability and performance of its implementation, Sbam (Spades BAsed Middleware) 
has ever been improved in [9]. Our goal is now to show the efficiency of the previously described QoS algo- 
rithm (Section |4]). We will focus on the size of the tree, and number of PIF that collaborate simultaneously. 
We will observe the behavior not only from the number of exchanged messages point of view but also in 
term of duration needed to performs CoPIFs. 



5.1 Sbam 

We use the term peer to refer to a physical machine that is available on the network. In our case, a peer is an 
instantiated Java Virtual Machine connected to other peers through the communication bus. We call nodes 
the vertices of the prefix tree. 

Sbam is the Java implementation of the Dlpt. Sbam proposes 2-abstraction layers in order to support 
the distributed data structure: the peer-layer and the agent-layer. The peer-layer is the closest to the hardware 
layer. It relies on the Ibis Portability Layer (IPL) (17] that enables the P2P communication. We instantiate 
one JVM per machine, also called peer. JVM communicate all together as a P2P fashion using the IPL 
communication bus. The agent-layer supports the data structure. Each node of the Dlpt is instantiated as 
a Sbam agent. Agents are uniformly distributed over peers and communicate together in a transparent way 
using a proxy interface. Since we want to guarantee truthfulness of information exchanged between Sbam- 
agents, the implementation of an efficient mechanism ensuring quality of large scale service discovery is 



quite challenging. In the state model described in the section 3.2 a node has to read the state and the variables 
of its neighbors. In Sbam, the feature is implemented using synchronous message exchange between agents. 
Indeed, when a node has to read its neighbor states, it sends a message to each and wait all responses. 
Despite the fact that this kind of implementation is expensive, especially on a large distributed data structure, 



experiment (Section 5.6 1 shown that our model implementation stays efficient, even on a huge prefix tree. 



5.2 Experimental platform 

Experiments were run on the Grid' 5000 platforrrfl[7], more precisely on a dedicated cluster HP Proliant 
DL165 G7 17 units, each of them equipped with 2 AMD Opteron 6164 HE (1.7GHz) processors, each 
processor gathering 12 cores, thus offering a 264-cores platform for these experiments. Each unit consists 
of 48 GB of memory. Units are connected through two Gigabit Ethernet cards. For each experiment, we 
deployed one peer per unit. 



http: //www.grid50 00 . f r/ 
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Fig. 5. CoPIF behavior 



5.3 Scenario of experiments 

The initialization of an experiment works in three phases: (?) the communication bus is started on a com- 



puting unit (Section 5.2 ), (ii) 16 peers are launched and connected together through the communication bus, 
and (Hi) a pilot is elected using the elect feature of the communication bus. 

After the initialization, the pilot drives the experiment. It consists in two sequential steps. First, it sends n 
insertion requests to the distributed tree structure. An insertion request leads to the addition of a new entry in 
the Dlpt tree (Section[3]). The insertion requests are sent to a random node of the tree and routed following 
the lexicographic pattern to the targeted node. Doing so, the node sharing the greatest common prefix with 
the service name is reached. If the targeted label does not exist, a new node is created on a randomly chosen 
peer and linked to the existing tree. 

In the final step, the pilot selects a set of nodes to initiate classic-PTFs and CoPIFs (Section|4]). In order 
to observe distributions, 10 replications of this basic scenario are executed. 



5.4 Failures 

At this level of description we can distinguish two kind of failure: (i) failures in the Dlpt data structure, 
when the prefix tree data structure is corrupted; (ii) failures in the CoPIF state variables, when state variable 
dedicated to CoPIF feature are corrupted. 

In our experiments, we consider that the Dlpt data structure is not corrupt. It corresponds to the worst 
case in term of CoPIF truthfulness check. Indeed, if the entire Dlpt data structure is correct, the CoPIF 
has to explore the entire Dlpt data structure to check it. 

Next, remind that the objective of our experiment is to evaluate the efficiency of CoPIF compared to 
classic-PTF. So, we consider that the CoPIF state variables are not corrupted and we measure the number 
of messages and the duration of CoPIF when the self-stabilization CoPIF has converged, it means after the 
clean part of the CoPIF state variables. 



5.5 Parameters and indicators 



The experiments conducted are influenced by two main parameters. First, n denotes the number of inserted 
services in the tree. Second, k refers to the number of PIF waves that are collaborating together. 



16 In these experiments, three trees were created with n in the set {2500, 10000, 
40000}. The number of PIF that collaborate (k) was taken from the set {1,2,4,8, 

16, 32, 64}. For each couple (n,pif), 10 replications are performed. Thus, 210 experiments were conducted. 

Strings used to label the nodes of the trees were randomly generated with an alphabet of 2 digits and a 
maximum length of 18 (in a set of 524287 key). 

For each experiment we observe two indicators: (i) the total number of exchanged messages observed 
and (ii) the time required to perform k classic-PlFs or CoPIFs over distributed data structure, i.e., the time 
between the issue of the PIFs and the receipt of the response on all nodes that initiate PIFs. 

For each indicator we obtain 21 sets of 10 values. We present evolution of the median-value, of the 10- 
replication according to k, the number of PIFs (Figures 5(a) and 5(b) I. The comparison of these indicators 
for classic-PlFs and CoPIFs provides us a qualitative overview of the gain obtain using CoPIF. 

In order to quantitatively evaluate the efficiency of the CoPIF strategy, for an indicator (/) and for a 
given number of PIFs k, we compute the efficiency criterion with the following formula: 

F _ hnd.pif 

Plf X /coll,/,!/ 

where /classic,?;'/ (resp. IcoPW.pif) is the value of the indicator / for a given k and in an classic-PlF (resp. 



CoPIF) context. The evolution of this efficiency criterion are shown in Figures 6(a) and 6(b) 



5.6 Results 



Figure 5(a) (resp. 5(b)) presents the evolution of the number of messages (resp. duration) needs to execute 



PIFs according to number of PIFs (k) that are simultaneously performed and the size of the data structure 
on which PIFs are performed. The y-axis represents the number of exchanged messages (resp. the duration). 
On these figures, classic-PlFs and CoPIFs strategies are compared. On both curve, the ;t-axis represents the 
number of PIF (k) that are simultaneously performed. On these figures, we present 2 curve triplets. Solid 
(resp. dashed) curves triplet describes indicator in an classic-PlF (resp. CoPIF) context. For each triplet, 
red-triangle-curve (resp. green— \— curve and blue-x-curve) describes behavior of indicator for n = 2500 
(resp. n = 10000 and n = 40000). 

When the indicators explode in for classic-PlF strategy, they stay stable for CoPIF strategy. It qualita- 
tively demonstrates gain of the CoPIF strategy over classic-PlF approach. The introduction of the efficiency 



criterion in Section 5.5 allows us to measure quantitatively this gain and its behavior. 



Figure 6(a) and 6(b) present the efficiency of CoPIF according to the number of exchanged messages 
and the duration. On these figures we want to observe the impact of the size of the data structure on the 



efficiency of CoPIF. Figure 6(a) reveals us that, in term of number exchanged messages, on small data 



structure, the collaborative mechanism is less efficient than on huge one. This result was expected because 
on small data structure the number of messages due to collision between collaborative PIFs (overhead) rep- 
resents a more important part of the entire number of exchanged messages. So, the bigger the data structure, 
the more efficient CoPIF. 



More interesting is the analyze of the Figure 6(b) Indeed we can observe the same tendency in term 



of duration but the efficiency decrease faster with the number of PIFs that are simultaneously performed. It 
is explain by the fact that overhead messages introduced by CoPIF are particularly expensive messages in 
term of duration. It quantitatively demonstrates gain of the CoPIF strategy over classic-PlF approach. 



6 Conclusion and Future Work 

In this paper we provide a self-stabilized collaborative algorithm called CoPIF allowing to check the truth- 
fulness of a distributed prefix tree. CoPIF implementation in a P2P service discovery framework is experi- 
mentally validate, in qualitative and in quantitative terms. Experiment demonstrates the efficiency of CoPIF 
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w.r.t. classic-PJF. CoPIF overhead represents a small part of the number of exchange messages and of the 
time spend, specially on huge data structures. 

We conjecture that the stabilization time is in 0{h 2 ) rounds and the worst case time to merge several 
classic-PIF waves is in 0(h) rounds, h being the height of the tree. We plan to experimentally validate this 
two complexities. Indeed, experiment were driven considering no corrupted CoPIF variables. In order to 
do that, we need to define a model of failure, implement or reuse a fault injector and couple it with Sbam 
before driving a new experiment campaign. 
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